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LOAD BALANCING TECHNIQUES FOR 
INTER-DOMAIN TRAFFIC ENGINEERING 

BACKGROUND OF INVENTION 

[01] The present invention relates to a method for the management of inter-domain 
traffic over the Intemet. In particular, the present invention relates to methods for 
selecting the most efficient and uncongested paths for inter-domain traffic. 

[02] A major challenge for inter-domain traffic engineering is the level of uncertainty 
an Intemet Service Provider (ISP) is facing when selecting paths for Intemet traffic. With 
many potentially competing ISPs, global coordination of Intemet traffic to select the best 
path and minimize congestion can be diflBcult. Moreover, a single ISP only has limited 
knowledge about user demand, available network resources, and routing policies at the 
peers. 

[03] One of the major problems encountered in inter-domain traffic engineering is 
appropriately reacting to congestion, losses, and/or delays experienced by traffic between 
a home network and other domains. This problem is encountered in a home network with 
multiple links into mother domain, such as large ISPs with multiple peering links to 
another backbone, or a smaller local network multi-homed to two or more backbones. In 
such cases, the company paying for these links may find that, under normal policies. 
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traffic exiting the home network through one of the Unks is experiencing congestion, 
while other Unks are under-utilized. In some cases, the congestion experienced by traffic 
leaving or entering the home network may actually occur outside its domain. In order to 
improve the performance experienced by the connections generating the associated 
5 traffic, the ISP operator needs to find a way to alleviate the congestion, losses and/or 
delays. 

[04] In simple cases, where the congestion is on the home network's egress link, 
congestion may be detected via Simple Network Management Protocol (SNMP) traffic 
measurements. SNMP is a set of protocols for managing complex networks which was 

10 first used in the 1980s. SNMP works by sending messages, called protocol data units 
(PDUs), to different parts of a network. SNMP-compUant devices, called agents, store 
data about themselves in Management Information Bases (MIBs) and retum this data to 
the SNMP requesters. A user can determine an alternate route for the traffic by using 
SNMP to measure the available capacity on the altemate egress links. Under such 

1 5 circumstances, there are any number of approaches to the problem of load balancing the 
traffic with the only real problem being the way it is implemented because typical inter- 
domain routing using Border Gateway Protocol (BGP) is a relatively coarse mechanism 
for traffic engineering. 

[05] A variety of metrics can also be used when a user is trying to determine the best 
20 path to use for sending traffic over a network. Some routing protocols, such as Routing 
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Information Protocol (RIP), use only one metric and that is hop count. And some routing 
protocols, such as Interior Gateway Routing Protocol (IGRP), use a combination of 
metrics. The metrics most commonly used are: (1) hop count - the number of routers that 
a packet must go through to reach its destination; (2) bandwidth - the data capacity of a 
5 link; (3) delay - the length of time to move the packet from the source to destination; (4) 
load - the amount of activity on a network resource; (5) reUability - the error rate of each 
network link; (6) ticks - the delay on a data link using IBM PC clock ticks; and (7) cost - 
an arbitrary value assigned by an administrator. The best route depends on the metrics 
and metric weightings used to make the calculation. For example, one routing protocol 
10 might use the number of hops and the delay, but might weigh the delay more heavily in 
the calculation. Thus, a route having more hops and shorter delays may be less expensive 
than a route having fewer drops and longer delays. Paths that are expensive to use are 
usually avoided. Such metrics are a useful tool but they cannot accurately account for 
traffic in other domains. 

1 5 [06] Leaving aside the practical problems of implementing a method for balancing the 
traffic load, there is a more fundamental problem. In some cases, the congestion 
experienced by traffic leaving or entering the home network may actually occur outside 
the domain so that the operator is unable to make direct observations of the traffic along 
different paths. Indeed, the operator's knowledge is limited to the home network and he 

20 or she does not necessarily know the routing, topology, or capacity of the network that the 
traffic traverses. However, the operator can often still detect congestion in the network 



using active probes, and the operator may want to take remedial actions to alleviate or 
avoid this congestion. 

[07] If the operator chooses to alleviate or avoid congestion that is detected on the 
network, he or she must balance loads across multiple paths without knowledge of the 
5 available capacity and the details of the paths themselves. The only information available 
to the operator are coarse measurements of statistics (such as loss rate) and Round Trip 
Time (RTT), which can be used to infer congestion, hi addition, the operator may be able 
to use some of the more recently developed techniques to estimate bottleneck bandwidth, 
for example, an approach based on network tomography. These mechanisms are unlikely 
10 to give an entirely accurate picture of the paths in question, but can narrow down the 
range of possibilities. 

[08] A further, and even more difficult problem, is that multiple operators may decide 
to use similar approaches to avoid congestion and may be simultaneously rerouting their 
traffic. In this case, they will play a kind of "game" against each other to optimize the 
15 utiUzation of the extemal domain, not knowing that other operators are changing traffic 
paths. Based on past experience, it has been found that such "games" (where individuals 
optimize their own utility) do not always lead to the global optimal behavior. 

[09] Accordingly, there is a need for a method which allows the network operator to 
ease congestion and to balance the loads on the network paths. More specifically, there is 
20 a need for a method which allows the operator to estimate the amount of traffic across 
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different paths in a network on a fairly accurate basis and to reroute some of the traffic to 
uncongested network paths. 

SUMMARY OF THE INVENTION 

[10] In accordance with the present invention, a method for balancing traffic across 
5 paths connecting a network to the Internet is provided. The method includes forming a 
connection between a home network and a large network which connects to a plurality of 
networks. The connection from the home network to the larger network includes a 
plurality of paths carrying traffic in the form of data packets and each path has a path 
load. One of the paths is selected from the plurality of paths which includes the selected 

10. path and other paths. The selected path has a traffic load and an initial overload. Each 
path connecting the home network to the larger network has a path load and the selected 
path has a traffic load and m initial overload. In a preferred embodiment, the home 
network is an internet service provider (ISP) and the larger network is the Internet. The 
method includes measuring the amount of traffic from the home network to the large 

15 network over the selected path, the congestion over the selected path and the available 
capacity over the selected path. The method also includes choosing the path load for each 
of the plurality of paths using a fractional allocation strategy, wherein the time to generate 
information is minimized and the amount of traffic lost to overloads is minimized and 
distributing a portion of the traffic from the selected path to the other paths based on the 

20 fractional allocation strategy. 
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[11] In a preferred embodiment of the present invention, the fractional allocation 
strategy includes the steps of: (a) associating the paths with a counter /, wherein the 
counter is a number equal to one (1) and there are a total of j paths; (b) calculating the 
total initial selected path overload; (c) calculating the selected path load, wherein the load 
5 is equal to the initial selected path overload less the sum of the low capacity boimdary for 
/ path(s); (d) calculating the portion of the traffic on the selected path to be distributed 
using a bi-sectional search strategy, preferably a multidimensional iterative bisection 
search algorithm; (e) distributing a portion of the traffic on the selected path to the other 
paths; and (f) stopping if there are no more paths (/ = j\ otherwise increasing the 
10 numerical value of the counter by one (1) and go to step (c). 

[12] In another preferred embodiment of the present invention, the portion of the traffic 
is 

distributed to the other paths using the equation 



15 wherein x,- is the path load, // is low capacity boundary, A, is high capacity boundary, P is 
the total number of paths and xo(0) is the initial overload on the selected path. 

[13] The cost for using a path connecting the home network to the larger network is 




(1) 



measured using the equation C = XXI^ 



[xi{t) - CiCf)]"*", wherein C is the cost, Tis the time 



period 
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oyer which the feasible solution is obtained, P is the number of paths between the home 
network and the large network, x is path load and c is the capacity of the path at time t. 

[14] Li preferred embodiments, the amount of traffic from the home network to the 
large network over the selected path is measured using flow level measurements or 
5 Simple Network Management Protocol (SNMP). The congestion over the selected path 
is measured using active probes, passive measurements of traffic details Transmission 
Control Protocol (TCP) Synchronize/Acknowledgement (SYN/ACK) response time or 
Round Trip Time (RTT), and loss measurements. And the available capacity over the 
selected path is measured using flow level measurements. Simple Network Management 
10 Protocol (SNMP) link measurements. Round Trip Time (RTT), loss measurements, 
active probes, or Transmission Control Protocol (TCP) Synchronize/Acknowledgement 
(SYN/ACK) response time. 

[15] The method of the present invention allows a local network operator, such as an 
ISP, to balance traffic loads across paths connecting the local network to the Internet and 
1 5 minimize congestion and improve efficiency. 

BRIEF DESCRIPTION OF THE FIGURES 

[16] Other objects and many attendant features of this invention will be readily 
appreciated as the invention becomes better understood by reference to the following 
detailed description when considered in connection with the accompanying drawings 



wherein: 

[171 FIG. 1 is a graph which illustrates the two-dimensional (2-d) case for the 
multidimensional iterative bisection search (MIBS) method. 

[18] FIG. 2 A is a bar graph showing a comparison of different search schemes based 
5 on convergence time when xq = 10,000 and © = 1 .00. 

[19] FIG. 2B is a bar graph showing a comparison of different search schemes based 
on total cost when jcq = 1 0,000 and g) = 1 .00. 

[20] FIG. 3A is a bar graph showing a comparison of different search schemes based 
on convergence time when xq = 10,000 and © = 1 .05. 

10 [21] FIG. 3B is a bar graph showing a comparison of different search schemes based 
on total cost when jcq = 10,000 and to = 1 .05. 

[22] FIG. 4A is a graph showing a comparison of different search schemes for varying 
numbers of paths based on convergence time whenj^o = 10,000 and (o = 1 .00. 

[23] FIG. 4B is a graph showing a comparison of different search schemes for varying 
15 numbers of paths based on total cost when xo = 10,000 and co = 1 .00. 

[24] FIG. 5 A is a graph showing a comparison of different search schemes for varying 
numbers of paths based on convergence time whenxo = 10,000 and co = 1.05. 
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[25] FIG. 5B is a graph showing a comparison of different search schemes for varying 
numbers of paths based on total cost whenxo = 10,000 and © = 1.05. 

[26] FIG. 6 A is a graph showing a comparison of different search schemes for varying 
numbers of paths based on convergence time when xq = ip,000 and (o = 1 .20. 

5 [27] FIG. 6B is a graph showing a comparison of different search schemes for varying 
numbers of paths based on total cost whenxo = 10,000 and (o = 1.20. 

[28] FIG. 7A is a graph showing a comparison of different search schemes for different 
prior estimates of the pinning interval, a, based on convergence time when xo = 10,000 
and CO = 1.01. 

10 [29] FIG. 7B is a graph showing a comparison of different search schemes for different 
prior estimates of the pinning interval, a, based on total cost when xq = 10,000 and oo = 
1.01. 

[30] FIG. 8 A is a graph showing a comparison of different search schemes for different 
available capacity for five (5) altemate paths based on convergence time when xq = 
15 10,000. 

[31] FIG. 8B is a graph showing a comparison of different search schemes for different 
available capacity for five (5) altemate paths based on total cost when xo = 10,000. 

[32] FIG. 9A is a graph showing a comparison of different search schemes for different 
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available capacity for ten (10) alternate paths based on convergence time when xq = 
10,000. 

[33] FIG. 9B is a graph showing a comparison of different search schemes for different 
available capacity for ten (10) altemate paths based on total cost when xo = 10,000. 

5 [34] FIG. 1 OA is a graph showing a comparison of different search schemes for 
different available capacity for twenty (20) altemate paths based on convergence time 
when xo= 10,000. 

[35] FIG. 1 OB is a graph showing a comparison of different search schemes for 
different available capacity for twenty (20) altemate paths based on total cost when xq = 
10 10,000. 

[36] FIG. 1 1 A is a graph showing a worst case comparison of different search schemes 
for varying numbers of paths based on convergence time when xq = 10,000 and co = 1 .00. 

[37] FIG. 1 IB is a graph showing a worst case comparison of different search schemes 
for varying numbers of paths based on total cost when xq = 10,000 and co = 1.00. 

1 5 [38] FIG. 12A is a graph showing a worst case comparison of different search schemes 
for varying numbers of paths based on convergence time whenxo = 10,000 and co = 1.10. 

[39] FIG. 12B is a graph showing a worst case comparison of different search schemes 
for varying numbers of paths based on total cost when = 10,000 and co = 1.10. 
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DETAILED DESCRIPTION OF THE INVENTION 

[40] The present invention is a method for balancing traffic on a network when the 
operator does not have complete information about the operating and performance 
characteristics of the network. In order to balance network traffic, it is desirable for the 
5 home network operator to be able to move at least some traffic to uncongested network 
paths to ease congestion. For the present application, the term "home network" refers to 
the network, such as an ISP network, for which the operator is balancing the traffic 
entering and exiting. 

[41] A major challenge for inter-domain traffic engineering is the level of uncertainty 
10 that the operator of an ISP faces. With many potentially competing ISPs, global 

coordination can be difficuh simply because of the size of the network. A single ISP only 
has limited knowledge about user demand, available network resource, and routing 
policies at the peers. In addition, the operator has limited control over how the traffic is 
routed extemal to its domain, the home network. However, the ISP operator can 
1 5 determine how (i.e., at what links) the traffic exits its domain by using various tools such 
as Border Gateway Protocol (BGP). The present invention utiUzes the capability to select 
home network exit links and provides methods for traffic load balancing in the case 
where the network service provider possesses limited knowledge about user demand, 
available network resources, and the routing policies used on network paths. In 
20 particular, the present invention provides algorithmic techniques for determining how 
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traffic loads are shifted from congested links to uncongested links with unknown 
available capacities. 

[42] In order to implement the method of the present invention, three types of 
measurements must be available to the home network operator: (1) the traffic 
5 contribution of the home network to any one path; (2) the congestion on the paths 
connecting the ISP to the network; and (3) the available capacity on a path. 

[43] The home network operator must be able to measure the traffic contribution of 
the home network to any one path. This can be done by the home network operator 
fairly precisely, at whatever level is necessary, either using flow level measurements, or 
10 SNMP link measurements when it is equivalent to path traffic. 

[44] The home network operator must be able to measure congestion (inferred from 
Round Trip Time (RTT), and loss measurements) using active probes, or passive 
measurements of traffic details such as the Transmission Control Protocol (TCP) 
Synchronize/Acknowledgement (SYN/ACK) response time. (TCP is one of the main 
15 protocols in TCP/IP networks. Whereas the IP protocol deals only with packets, TCP 

enables two hosts to estabUsh a connection and exchange streams of data. TCP guarantees 
delivery of data and also guarantees that packets will be delivered in the same order in 
which they were sent.) Such measurements will, by their statistical nature, contain false 
positives and false negatives which will have to be factored into the calculations. 
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[45] The home network operator must be able to measure the available capacity on a 
path. These measurement scan be obtained using flow level measurements, SNMP link 
measurements, RTT, loss measurements, active probes, or passive measurements of 
traffic details, such as the Transmission Control Protocol (TCP) 
5 Synchronize/Acknowledgement (SYN/ACK) response time. However, these 

measurements are less accurate for measuring path capacity than they are for measuring 
the traffic contribution of the home network and congestion. Therefore, they are only 
used to obtain approximate bounds on the available capacity of a path. A worst case 
estimate of the bounds on the capacity of a path would be [0,/,], where fi is the 
10 capacity of the first hop link, leaving the home network. 

[46] The method of the present invention is based on the assumption that there are P 
+ 1 egress paths from the home network and that 1 is congested. The paths are indexed 
by /e (0, 1, . . ., P}. On each path,/?,, at time U it is assumed that there is available 
capacity, c, (0, which can vary over time due to changes in cross-traffic along the path 
15 of interest. In general, the available capacity, c/, is not known, except for some bounds, 
Ciif) e ^/ (OIj obtained through measurements. For the methods described herein, 
this interval is called the pinning interval, a. 

[47] The traffic being sent to a specific path can be directly measured and it is 
denoted as jc, while, without the loss of generality, the congested path is denoted as path 
20 "0." It is assumed that the capacity over the congested path, co, is known precisely - the 
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problem can be simply generalized if not, by including xo in the processes below. 
Under such assumptions, a simple transformation allows the problem to be considered, 
where co 0 and xq xq - cq, so that xq is now the excess load that needs to be 
distributed in order to alleviate congestion. 

[48] The congestion indications are denoted by indicator functions, where 



[49] The method of the present invention uses an infinite buffer model in which the 
amount of traffic dropped is modeled as the excess load, di (t) = [xi{t) - Ci(t)]^^ where 
[. . .]"^ denotes the positive part. This condition can be relaxed if the appropriate 
measurements to take such details into account are available. However, such 
measurements (for instance packet traces) are rarely available in practice. 

[50] The above statement also assumes that the time intervals, ^ = 1, 2, 3, are 
sufficiently far apart that transient effects fi:om changes, for instance in the offered 
loads (i.e., the traffic sent to a path, jc, (0), need not be considered and that congestion 
indications are instantaneous, though in reality they will always involve some delay. 
Typically, this delay would be measured in seconds, whereas, the time scale of the 
algorithm of the present invention can be in the order of minutes. Li addition, the 
individual paths are assumed to be disjointed in the sense that they do not share 
bottlenecks. Otherwise, the paths sharing the bottleneck could be incorporated into 




1, if path / is congested at t, 
0, otherwise. 
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one, larger path. 
Statement of the Problem 

[51] The method of the present invention provides an iterative algorithm that uses 
congestion indication measurements alone to determine a feasible solution to a load 
5 balancing problem. If a load balancing problem does not exist, the method will find the 
minimal cost solution. A minimal cost solution is found to be feasible if 

Xiit)<Ci{t\ = 

where 7 is the time at which convergence occurs. The minimum average long-term 
10 cost solution is the solution that minimizes the average number of packets dropped over 
time and it can be expressed by the following expression 

lim 1 p 

[52] In addition, if there is a feasible solution, it would be desirable for this solution 
to satisfy a number of properties. For instance, the solution should have a short 
15 convergence time (the time, T, at which the feasible solution is obtained should be as 
short as possible, both 

in expectation, and mean) and the solution should drop as few packets as possible so 
that it has 
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a low cost, where cost is measured by C = ^^[a:»(0 -^'(O]^- 

[53] The method does not assume that path traffic equals link traffic on the egress 
links, so there can be more than one set of paths across which traffic can be balanced. 

The Static Case 

5 [54] The first case considered is the static case. In this case, it is assumed that: (1) 
the network, and cross-traffic loads do not change over time, so that c, (0 = c„ where c, 
is a constant; (2) the traffic is inelastic, and is not directly affected by changes in its 
routing, or the congestion level; and (3) the congestion indications on each path are 
always correct. 

10 [55] Li the limit where the discretization size 5 ^ 0, a feasible solution can always 

be found when xo < cv. As the discretization increases, there may be cases where 

this relationship is satisfied, but there is no feasible solution. Li order for a feasible 

solution to exist, a necessary and sufficient condition is that Xo < ^^.^^ [ci/S]x B , where 

[x] denotes the largest integer less than x. If the capacity is discretized with the same 
15 quantization, 5, as the traffic, then the two conditions are equivalent. 

[56] Where there is no feasible solution, the minimum long-term cost solution is any 
one for which all of the paths 1, . . . , P are congested. Any of these solutions will be 
equivalent with respect to long-term cost (as defined above). Since it is relatively easy 
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to find such solutions, these cases are not addressed. 

[57] When there is no feasible solution, a different cost metric is used. For example, 
it may be desirable to have congestion on as few paths as possible to reduce the number 
of users impacted by congestion, or it may be more desirable to have moderate levels of 
congestion on all paths, rather than a single path with high congestion. However, 
before alternate approaches are discussed, the solution for the static case is found. 

[58] Under the conditions for the static case, the bounds of the capacity estimates can 
be tightened after each time interval. For example 

lit + 1) = max {lit\ Xi{t)}, Qit) = 0, 

hiit+\) = min {hi(t), Q(0 = 1. 

If the path is congested, then the available capacity on that path is less than the 
currently used capacity, whereas, if the path is uncongested, the available capacity is at 
least what is currently being used. In the discrete case, slightly better bounds can be 
obtained by taking 

hi (t+l) = min {hit), xit) - B} , Qit) = 1 , 
because x, can only be changed in multiples of B, 

[59] The problem is to choose the traffic sent to a path (also referred to herein as 
traffic path load or path load), jc, (t), at each time interval, t, so that the total load to be 
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distributed, ;co (0), is spread across paths 0, . . . , P, and converges (with as low a cost as 
possible) to a feasible solution, one in which Xi(t) < Ci{t\ V/ . By choosing G {hit 

+ 1), hi (t + 1)), it is possible to generate additional information that can be used in the 
next iteration to refine the bounds for the possible capacities, c,. Either the low 
5 capacity boundary, // , can be increased, or the high capacity boundary, hi, can be 
decreased. 

[60] The problem is how best to choose the traffic load sent to a path, x/, so as to 
generate information as quickly as possible, while at the same time reducing the 
probability that traffic will be lost to overloads. Several different strategies can be used 
10 to approach this problem and they are described below. In each case, the algorithm is 
terminated at the point when either a feasible solution is reached, or all links are 
congested. Convergence is guaranteed by the fact that the allowable region [/,, hi] will 
decrease at each time step as long as at least one Xi C {U, hi) is chosen. 

Proportional Allocation 

15 [61] One embodiment of the present invention approaches the problem of choosing 
the traffic load sent to a particular path by trying to minimize the expected losses at 
each iteration of the algorithm. Given no other information, the simplest assumption 
that can be made is that the available capacity on each path is uniformly distributed 
between the bounds, that is, c, ~ U [// (0, hi (t)], at some time, t. In fact, each 

20 measurement may be considered to provide more information than simply the bounds. 
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Therefore, the most reasonable distribution to use might be something other than the 
uniform distribution. A preferred embodiment of the present invention uses the 
following allocation of Xi 

hi-li ( ^ ^ 

Xi = h + — — X ^o(O) - ^/i , (1) 

5 This method is referred to in the present invention as "proportional allocation." 

The reason for selecting the name can be seen in the simple case where h = 0, V/ . In 
this case, the allocations to each path will be proportional to the high boimdary, hu for 
that path. 

[62] In the discrete case, the path loads, jc/, must be multiples of the block size B. 
10 The algorithm can take several forms, but a simple approach is to take the Xi-B 

\xil B\^ and then to take the remaining load, and distribute it in a round-robin fashion 

between the paths. 

[63] The final algorithm then proceeds in a series of steps. The initial algorithm uses 
coarse bandwidth estimates for the low, //(O), and high, /2/(0) boundaries, and then 
15 proceeds by choosing the path load, x/, as in equation (1). Once congestion has been 
measured on each path, the bounds are updated, and the steps are repeated, until either 
a feasible solution is reached, or all paths are congested. Although this approach 
minimizes the expected cost at each step, it does not minimize the overall cost. Such a 
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minimization also needs to reduce the boundary intervals [//, hi] as quickly as possible. 
Fractional Allocation Strategies 

[64] In the proportional allocation method discussed above, it was assumed that the 
initial path load was zero, xq = 0, in order to minimize the cost at each step. Otherwise, 
5 A = - 1 , and hence jc, = hi, which yields a value for xo that is not feasible. 

[65] In another embodiment of the present invention called the "Fractional 
Allocation Strategy," an altemative method is used which does not assume xq (t) 
= 0, > 0, but instead assigns only part of the initial path load, atq, to the other paths at 
each step. Fractional Allocation Strategy corresponds to the strategy where the initial 

10 overload on path "0" (jco(O)) or the "selected path" is broken up into smaller segments 
and once a starting segment size (to be off-loaded and distributed to the other paths 
connecting the home network to the larger network) has been decided, the subsequent 
segment sizes are decided based on some rule (e.g., linear, bisection etc.). However, 
the segments bandwidth can be distributed amongst users according to any criteria, 

15 with Proportional Allocation used as the default rule. Algorithm 1, which is shovm 
below, describes the Fractional Allocation scheme. 

Algorithm 1 — Fractional Allocation Strategy 

20 while ! (Feasible Allocation) do 

Update Load as 
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Load(0 = xo{0) -Y,^i{t) 
1=1 

Calculate traffic to be distributed 

yit) < Load(0 

according to linear search, bi-section search or exponential search 

5 strategy 

Distribute amongst paths using equation (1) 

(with ( xo(0) - J]!" J // ) replaced by y (t)) 

Update pinning intervals [/, + 1), hi (t + 1)] 
end while 
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[66] Three strategies can be used for choosing the traffic to be distributed (y(t)): (1) 
linear search, (2) bi-section search, and (3) exponential search. 

[67] The "linear search" strategy has a parameter S TEP.S IZE which is chosen by 
15 setting a number of roimds, N, that it will take for the algorithm to converge. Hence, 



STEP SIZE = 



N 



At each iteration, the traffic to be distributed, = min{STEP_SIZE, Load(0} of the 
load is assigned to the paths / = 1,. . P. 

[68] As the algorithm progresses, the traffic being reallocated at each step, y(t), tends 
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to be allocated to uncongested links, and so the lower bounds, /,-, on uncongested paths 
will increase, thereby decreasing the load on path "0" by approximately the 
STEP_SIZE at each interval, and converging in after about steps. Essentially, the 
aim of this approach is to minimize the congestion on paths, / = 1,. . P. However, this 
5 strategy allows ongoing congestion on path "0" to persist for longer periods of time. In 
some cases, it may be more desirable to allocate traffic this way, for example, when it 
is necessary to reduce the overall number of dropped packets and to keep the number of 
congested paths to a minimum at all times. 

[691 The "bi-section search" strategy is the preferred strategy of the present 
10 invention. The "bi-section search" strategy takes a parameter 0 < a < 1, which would 
typically be 0.5 (thus, giving the search strategy its name) and then performs the 
algorithm defined by choosing the traffic to be distributed, y{t), that satisfy 



This approach is intended to allocated the traffic more quickly at the start, and then 
15 slows down as the bounds become more narrow. Accordingly, this approach would be 
expected to converge more quickly than the linear search. 

[70] An alternative is the "exponential search" strategy, which is defined by three 
parameters: EXP_BASE (chosen to be either constant, or a constant fraction of Load), 
EXP_F ACTOR, which is (by default) 2, and ROUND_ID. The traffic to be 




distributed,j;(0, are chosen using 

y{t) = min {EXP_BASE x EXP_,FACTOR^^^^-*^ Load(0} 
Multidimensional Iterative Bisection Search 

[71] In a preferred embodiment of the present invention, a "multidimensional 
5 iterative bisection search" (MIBS) is used wherein the entire initial congested load 
from path "0" is allocated at each step. This approach is analogous to a traditional 
binary search, though it is generalized to a higher dimensional, constrained space. 

[72] To illustrate the MIBS approach, the simplest, non-trivial case, is considered, 
where there are P = 2 altemate paths across the network boundary which can be used to 
10 balance the excess load. The graph in Figure 1 illustrates this two-dimensional (2-d) 
case. The graph shows the capacity constraints, xi < Ci and X2 < C2, (vertical and 
horizontal dotted lines, respectively), and the constraint, xi + X2 = xq, is plotted as a 
dashed line (the constraint line). The feasible solutions fall on the small solid segment 
of the constraint line. 

15 [73] The multidimensional iterative bisection search approach can start with zero 
knowledge of the bounds (i.e., effectively it is assumed that the low and high 
boundaries equal zero (// = 0 and hi = xo(0)) for all When better information is 
obtained about the initial pinning interval(a), it can be easily exploited. The approach 
is initialized by using the proportional allocations, which in this (zero knowledge) case 
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assigns equal traffic to paths 1 and 2. The approach proceeds much as a typically 
binary search would proceed. A new set of pinning intervals is selected based on the 
constraints that are satisfied. The pinning interval, a, (along the constraint line) is then 
bisected and tested again. This is shown by the series of dots in Figure 1 . 

5 [74] The great advantage of the local search approach is that it provably converges to 
the feasible region very quickly. Its worst case performance (for the 2-d case) isT= 
log2(l/Z), where L is the proportion of the constraint line (in the positive quadrant of 
IR\ that is feasible. In cases where the solution must be discretized, the smallest 

feasible region (if one exists) is I ~ 5/^2xo(0), and so the convergence time will 

10 increase for larger initial loads, and decrease for larger discretizations (assuming the 
available capacities are scaled appropriately). 

[75] In fact, for the two-dimensional case, if the total available capacity, c = 2^.^^^/ , 

is no more than the load to be distributed, the multidimensional iterative bisection 
search is provably optimal. This can be shown by the case where c = xo(0). In this 
15 case, a standard binary search with its normal properties is simply being performed. 

Algorithm 2 — Multidimensional Iterative Bisection Search 

NUMJTERATIONS: Number of Local Iterations 
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while! (Feasible Allocation) do 

Allocate Rates Proportionally amongst all paths 
1 = 0 

while (I + + < NUMJterations) do 
5 Shuffle the list of paths. 

while (! End of List of Paths) do 

Pick an unpicked pair from the List, (Let paths i, j be 

picked.) 

Sij Xi + Xj 

10 LOi = max(li, sy - hj) 

HIi = min(hi, sy - Ij) 
LOj = Sij-HIi 
HIj = Sij -LOi 
Xi = L0i + (HIi-L0i)/2 

15 Xj ^ Sij ^ Xj 

end while 
end while 
end while 
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[76] The disadvantage of the two-dimensional approach is that it is sUghtly more 
compHcated to generalize the algorithm to higher dimensions. In higher dimensions, it 
is necessary to run the equivalent of the binary search on a P - 1 dimensional manifold 
embedded in the positive quadrant of R^. The algorithm discussed in the next section is 
25 a good approximation to this search. 
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Modified Multidimensional Iterative Bisection Search 

[77] Another embodiment of the present invention, is a modification of MIBS. In 
the two-dimensional case, MDBS is provably optimal if c < x^^O). However, it is no 
longer optimal if c > a:o(0). The modified multidimensional iterative bisection search 
(MMIBS) shown below is provably optimal (in the 2-d case) when the total capacity is 
given. It is also provably better than MIBS when a true lower bound of the total 
capacity is known. When c < a:o(0), MMIBS behaves exactly as the MIBS algorithm. 

Algorithm 3 - Modified Multidimensional Iterative Bisection Search 



if (know true TotalCapacity) then 
estCapacity = TotalCapacity 

else 

estCapacity = max(totalDemand, capacityLowerBound) 

end if 

extra if 

extraCapacity = estCapacity - totalDemand 
Initialize /, = 0 and hi = estCapacity 
NUM-ITERATIONS: Number of Local Iterations 
while ! (Feasible Allocation) do 

Allocate Rates Proportionally amongst all paths. 

1 = 0 

while (!++<NUM_ITERATIONS)do 
Shuffle the list of paths, 
while (!End of List of Paths) do 
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Pick an unpicked pair from the List, (Let paths i j are 



picked.) 



Sy Xi'^ Xj 

S = Sij + extraCapacity 
5 LOi = maxQiyS - hj) 

HIi = min(hi,S - Ij) 
LOj^S-HIi 
HIj = S-LOi 

Xi = LOi + {HIi - LOi - extraCapacity)/! 
10 Xi = max(min(xi^Sij)fi) 

Xj — S ij ~ Xi 

end while 
end while 
end while 
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EXAMPLES 

[78] The examples set forth below serve to provide further appreciation of the 
invention but are not meant in any way to restrict the scope of the invention. 

20 [79] The examples illustrate the performance of the different approaches for the 
static case. The results were generated using simulations in which the true available 
capacities on paths z = 0,. . P are chosen randomly by choosing a set of random 
variable z, - C/(0,1), (that is, uniformly on the interval (0,1)), and then setting the 
capacities to be 

27 



ZiXo(O) 

C/ = (OX— — , 

where the number of available paths, co is a tunable parameter in order to allow the size 
of the feasible region to be varied. In the continuous case, when the number of 
available paths equals one (o = 1), there is exactly enough capacity to allow all of the 
5 traffic to be carried; when the number of available paths is greater than one (co > 1) a 
feasible solution exists, and when the number of available paths is less than one ( O) < 
1), no feasible solution exists. In the discrete case, the capacity, c, ,is also discretized, 
so that the niunber of available paths (co) no longer has the same strict relationship to 
the feasibility of the solution. 

10 [80] Where not otherwise specified, initial estimates are generated for /, = 0 and h i 
by a random variable distributed between the maximum capacity of any path, and the 
load to be distributed, for example 

hi^h--U (Cmax,^o(0)), V/ = 

[81] Results for different allocation strategies are presented as the number of 
15 available paths (co) and the accuracy of initial measurements of pinning intervals (a) 
vary. The results are also presented for the worst case performance of these allocation 
strategies. The performance metrics of interest are convergence time and total cost 
incurred. The results presented in the examples show the convergence times and total 
costs over 1,000 simulations. 
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[82] In all of the results, the discretized version is used, with 5=1, and the initial 
load JCo(O) is varied rather than B. An equivalent approach would be to fix xo(0), and 
vary \IB in the same way. 

EXAMPLE 1 

5 Comparison of Various Allocation Strategies 

[83] This example compares the different allocations strategies: proportional 
allocation, linear-search, bisection search, exponential search and multidimensional 
iterative bisection search (MIBS). Figures 2A-B and Figures 3A-B show comparisons 
of these strategies, through both the convergence time and the total cost. The graphs 

10 show box plots giving the mean, quartiles, and extreme values of the 1,000 simulations, 
for a range of values for the number of available paths (co). The results clearly show 
that the linear and exponential approaches are much worse than any altemative. This is 
because these schemes are conservative in their approach, i.e., they do not cause 
congestion on any uncongested path, and thus incur high cost by not easing the 

1 5 congestion on the congested path. 

[84] For a small number of available paths ( co ), the bisection approach is also 
significantly worse than the proportional allocation and local search approach, which 
have very similar results. Given the additional complexity of the bisection approach, 
over straight proportional allocation, it does not provide any advantages. Also, the 
20 performance of MIBS is the same or marginally better than that of proportional 
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allocation. Therefore, the MIBS and proportional allocation strategies are preferred. 

EXAMPLE 2 

Comparisons of Different Allocation Strategies for Variable Number of Alternate 

Paths 

5 [85] This example studies the proportional allocation and the multidimensional 
iterative bisection search (MIBS) approaches. In this example, the total number of 
alternate paths that are available varied from 2 to 20 for a constant path load, xo(0) = 
10,000. The number of available paths, oo, also varied from 1.0 to 1.2. Figures 4 to 6 
show the resuhs - giving the average values as points, and the 95 percentiles of the 

10 distribution of results as vertical bars. The results indicate, both algorithms have similar 
performance vis-a-vis convergence time. However, MIBS outperforms the other 
strategies when the total cost is considered as a performance metric, particularly for large 
numbers of paths. Also, it is interesting to note that the cost and the convergence times 
increase as the total number of available paths increases, except for the MIBS with 

1 5 respect to cost, which reaches a peak, and then stays there. 

EXAMPLE 3 
Accurate Measurements of the Pinning Interval 
[86] This example compares the approaches to allocation when more accurate 
measiu-ements for the initial pinning interval, a, are available. In the simulations, this 
20 was tested by drawing the lower and higher thresholds of the pinning interval as 
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li(0)eU[(l'a)ciCj and 
hi(0)eU[ciCi(l-a)J 

for different values of a. Specifically, the pinning interval, a, was varied from 0.05 to 1. 
Figure 7 shows the results. The proportional allocation strategy converges faster than 
5 MIBS though the difference in convergence times is less than 1 iteration. Also, the 
results suggest that convergence times as well as total costs decrease as the value of the 
pinning interval, a, decreases. This is intutitive as a decreasing value of pinning intervals, 
a, suggests that we have more accurate estimates of the pinning interval, a, especially the 
lower boimds. 

10 EXAMPLE 4 

Performance of Different Allocation Strategies for Different Values of Available 

Capacity 

[87] This example presents the results for the performance of the proportional and 
multidimensional iterative bisection search (MIBS) strategies for different values of 

15 available capacities. For all the experiments, the total initial load, xo (0), was maintained 
at a constant value of 10,000 and the total available capacity was varied fi-om 10,000 to 
12, 500 by varying the number of available paths, ©. The total number of paths available 
was also varied from 2 to 20. However, Figures 8A-B, 9A-B and lOA-B show the results 
for only 5, 10 and 20 paths. These figures show that, as the available capacity increases, 

20 the convergence time and the total cost incurred decreases. This is because greater 

available capacity implies a bigger feasible region and, thus, it is easier and faster to find 
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a feasible solution. Figures 8 to 10 also show that, as the available capacity increases, 
MIBS outperforms the proportional scheme. Moreover, the gains are more significant if 
the number of available paths is larger. 

EXAMPLE 5 

Focused Capacity Performance of Different Allocation Strategies 
[88] In this example, the performance of the three schemes was compared when the 
available capacity is focused on one altemative path. The conditions for the tests 
included a constant initial load (xo (0)) and a given number of available paths (P), with a 
total available capacity c = xo (0), P - 1 paths with capacity q = 1, and one path with q = c 
-n + I units of available capacity. Figures 1 lA-B and 12A-B show the results of these 
tests. 

[89] Thus, while there have been described the preferred embodiments of the present 
invention, those skilled in the art will reaUze that other embodiments can be made 
without departing firom the spirit of the invention, and it is intended to include all such 
further modifications and changes as come within the true scope of the claims set forth 
herein. 



32 



